Dynamic adjustment of cloud game data streams to output device and network quality

ABSTRACT

Aspects of the present disclosure relate to systems and methods for the dynamic adjustment of data streamed over a network which is then displayed on an output device. In particular, aspects of the present disclosure relate to systems and methods for determining the display capabilities of an output device and then formatting two or more streams of data configured for display on the output device, wherein the user interface and video data are contained in separate streams, such that the content is displayed in a manner consistent with the display capabilities of the output device. The output can then be adjusted or prioritized based on device orientation and network quality.

FIELD

The present disclosure relates to the dynamic adjustment of datastreamed over a network. In particular, aspects of the presentdisclosure relate to systems and methods for determining the displaycapabilities of an output device and then formatting two or more streamsof data configured for display on the output device.

BACKGROUND

With the increasing prevalence of digital streaming services and variouscloud-based computing solutions, the ability to quickly and accuratelytransfer large amounts of data between remote devices is a criticaltask. Currently, data streaming services do not take into account thespecifications of the device that the data is meant to be displayed on.As a result, scaling and pixel variation between devices can createundesirable display scenarios, for example, text being unreadable on asmaller screen, or an aspect ratio of a display may not be suitable fora certain piece of media.

Additionally, digital streaming services and cloud-based computingsolutions may experience limitations in the quality and bandwidth ofnetworks established or used during the transfer of data between remotedevices when utilizing applications that are sensitive to latencies,such as video games. These limitations may lead to delays in the datatransmission and can thus cause latency, which typically createsinconsistencies during the use of an application. While client deviceswill attempt to achieve the lowest latency through a variety of methods,inevitably, each client device will experience a different amount oflatency due to differences in factors such as the decode speed oftransmitted data, render rates, input polling, or even the client'snetwork connection. In some forms of media, the latent display ofspecific media such as text or user interface (UI) data may be moredetrimental to the enjoyment of the user, causing user frustration andperhaps even abandonment of a title or the platform altogether.

Furthermore, on a wide variety of media playing devices available toconsumers, the orientation of the display may rapidly change dependingon the needs of the user. Certain device orientations may make viewing amedia title, which would otherwise be normally viewable in analternative orientation nearly impossible. Accordingly, there is a needin the art to find alternative means for adjusting and displaying mediastreamed over a network that takes into account network latency, displayorientation, and the specific display capabilities of a wide variety ofdevices.

SUMMARY

In accordance with certain aspects of the present disclosure, anon-transitory computer readable medium may contain computer readableinstructions embodied therein. The instructions may be configured toimplement a method when executed. The method may include determining thedisplay capabilities of an output device. Two or more data streamsconfigured for display on the output device may be established andformatted. The data streams may include separate streams for video dataand user interface (UI) data. The data streams may then be delivered toan output device.

In accordance with certain implementations of the present disclosure, amethod for streaming data to an output device may include determiningthe display capabilities of the output device. Two or more data streamsconfigured for display on the output device may be established andformatted. The data streams may contain separate information for videodata and user interface (UI) data. The data streams may then bedelivered to the output device.

In accordance with certain implementations of the present disclosure, acomputing system may include at least one processor unit, and at leastone memory unit coupled to the at least one processor unit. The at leastone processor unit and the at least one memory unit may be configured toperform a method. The method may include determining the displaycapabilities of an output device. Two or more data streams configuredfor display on the output device may be established and formatted. Thedata streams may contain separate information for video data and userinterface (UI) data. The data streams may then be delivered to theoutput device.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present disclosure can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 is an example of a screen shot of a portion of a media titledisplayed on an output device derived from data streamed over a network,and is intended to illustrate various components of displaying a title.

FIG. 2 is a flow diagram of an example system in accordance with certainaspects of the present disclosure.

FIG. 3 is a block diagram of an example system in accordance withcertain aspects of the present disclosure.

FIG. 4 is a flow diagram of an example asset management technique fromthe server side in accordance with certain aspects of the presentdisclosure.

FIG. 5 is a flow diagram of an example asset management technique fromthe output device side in accordance with certain aspects of the presentdisclosure.

DETAILED DESCRIPTION

Although the following detailed description contains many specificdetails for the purposes of illustration, anyone of ordinary skill inthe art will appreciate that many variations and alterations to thefollowing details are within the scope of the invention. Accordingly,the illustrative implementations of the present disclosure describedbelow are set forth without any loss of generality to, and withoutimposing limitations upon, the claimed invention.

Introduction

Aspects of the present disclosure relate to systems and methods for thedynamic adjustment of UI and video streaming data in response to varyingdevice types, orientations, and network quality.

In accordance with certain aspects, a client device configured tooperate on a network may provide a user with a list of one or moredigital assets that can be borrowed from a providing user. The may thenbe able to request the use of an asset that can be borrowed from aproviding user. The user may then receive certain rights, such as accessto an application or application features, from the providing user.Alternative embodiments provide a method in which a providing user maygrant asset rights to another user without first receiving a request.

Implementation Details

In FIG. 1, an example is provided of a screen shot of a media titledisplayed on an output device derived from data streamed over a network10. This example is intended to illustrate the various components of astreamed title. For example, on the output device 12, various portionsof the display may vary in quality and clarity depending on theorientation of the device or the quality of the network connection. Forexample, video data such as 14 may be clear when displayed independentof the device upon which it is displayed. However, UI data such as text16 and map data 18 may be difficult to display properly on a smallerscreen, or may be rendered unreadable or unusable when formatted to fita smaller display.

Turning now to FIG. 2, an illustrative example is provided of how amedia title may be displayed on various devices from data streamed overa network 100. In particular, FIG. 2 depicts an example process flow forproviding a varying output devices with several data streams inaccordance with certain implementations of the present disclosure. It isnoted that the example method of providing an application asset in FIG.2 may have one or more aspects in common with the methods indicated inFIGS. 4 and 5. It is also important to note that the example of FIG. 2is only a simplified example for purposes illustrating only certainaspects of how data streams may be provided in accordance with thepresent disclosure.

At the outset, it is important to note that in this example, a datastream is provided to an output device that is requesting a stream froma server containing media information, but data may be requested fromand delivered by alternative sources. In this example, as shown in FIG.2, a user utilizing an output device 103 requests the stream of a mediatitle from a server 102. The server receives this request 182 anddetermines the capabilities 186 of the output device 103 from a list ofcompatible devices. Relevant capabilities may include parameters such asscreen size in pixels, physical screen dimensions, screen resolution,color format (e.g., 8-bit, 15-bit, 16-bit, 24-bit, 30-bit, 36-bit, or48-bit color), and the like.

Additional examples of parameters which may be of use include, but arenot limited to the following: pixel aspect ratio (a pixel is typicallyassumed to be square with the same width and height. This is notnecessarily true especially for old SD television signals); color spaceinformation (RGB, YUV, etc.), gamma correction information, screenorientation, display backlight level (‘brightness level’), refresh rateof the display. With respect to refresh rate, is may be useful to knowthe maximum refresh rate (60 Hz), but the device may also support loweror even variable refresh rates.

Some of these parameters could change over time. For example, forhandheld devices such as tablets and smartphones, orientation maychange. It is also possible for the display capabilities to change whenstreaming to one device that is locally connected to a different displaydevice. For example, a user who is streaming video to a device such as atablet or phone may suddenly connect the device to a 4K display. Thiscan change some of the capabilities.

There may also be some other relevant graphics processing unit (GPU)parameters. For example some GPUs (especially on embedded platforms)have support for video overlays. Specifically, a frame may have multipledrawing layers, which are somehow combined (‘composited’) before theframe is output to the display. This can allow for independent renderingof video and UI. For example the video stream could be rendered on adifferent layer than the UI. The UI and video streams can have differentupdate frequencies and both don't have to ‘wait’ for each other. Forexample, digital televisions use overlays to layer a system menu systemon top of the ‘TV image’.

In alternative embodiments, the server may request the deviceinformation 183 from the output device when the display capabilities areunknown, and the output device may in turn deliver that information 184to the server. For example, the output device may include one or moresensors, e.g., an inertial sensor and/or user-facing camera, that allowthe output device to sense the orientation of a display screen relativeto a user. The display screen may be rectangular but have a non-squareaspect ratio (ratio of width to height). The server 102 may request thatthe output device indicate whether it is in a horizontal or verticalorientation relative to the user so that the data to be streamed may bebest formatted for presentation at that orientation.

Information regarding device capabilities (e.g. whether device supports480p, 720p . . . ) give the server information on what is technicallypossible. It would also be useful for the server to know certain currentsettings of the device such as orientation and output resolution. Otherinformation such as color depth may be less important, e.g., because itcan be adjusted at the client device or the server would dictate thisinformation for compression efficiency reasons or to reduce requiredbandwidth.

The server 102 may also request information about the quality of thenetwork 100 between the server and the output device 103. Suchinformation may include parameters such as network latency, availablenetwork bandwidth, packet loss, or network protocol. This informationcan be used in formatting the data streams 188. For example, somenetwork protocols, e.g., Uniform Datagram Protocol (UDP) are highlyunreliable. If the server 102 knows that the network 100 uses UDP thestreams 188 can be configured to include Forward Error Correction (FEC)packets that the output device 103 can use to recover lost data packets.

Streaming technology often uses FEC techniques. Before startingstreaming to a client device a server sets a bandwidth budget, e.g., 8Mbps for the stream. To set the bandwidth budget it is useful for theserver to pick some client device settings such as video resolution,frame rate and compression settings.

During streaming the server can monitor bandwidth and based on networkparameters such as bandwidth and packet loss and adjust our videoresolution, frame rate and compression settings accordingly. Similarsettings can be adjusted for audio streaming, which may become moreimportant. Knowing the audio settings allows the server to optimize theaudio stream for, e.g., a six channel surround sound setup (e.g., 5.1surround sound) or a stereo setup.

Some streaming technology reduces frame rate or resolution somewhatcrudely. For example, some video game streaming systems capture videofrom HDMI, so if a game is generating 60 fps and the streaming rateneeds to be reduced to 30 fps, the streaming server simply throws framesaway. In addition if resolution needs to be reduced, the server can justscale the video frame received from the game. By way of example, thegame may be generating video at 720P and if the server 102 needs tostream at 540P, the frame is just scaled in software.

Servers for some applications, such as online video games, often run theapplication software on different hardware than the hardware used tohandle streaming of video to the client devices. The streaming serverand application hardware may be located in a data center and connectedby a local area network. According to aspects of the present disclosure,formatting the video data could be done using the application thatgenerates video frames that are used for the video stream. By way ofexample, and not by way of limitation, the streaming server could notifythe hardware running the application that it needs to reduce frame rateor display resolution. The application can then adjust the frame rate orresolution or otherwise adjust the video formatting for the outputdevice 103. The application that generates the video frames often can doa better job of adjusting frame rate, display resolution or other videoformatting parameters. In addition, formatting the video this way couldeven save a little bit of electrical power (a lower resolution orframerate is less intensive for a server and hence uses less power).

Once the device display capability information is determined, the servermay create separate streams for the video and UI information 188. Thisstep is performed so that the UI and video data information can beformatted separately on the output device 103, allowing for the separatescaling of UI and video data to ensure that all aspects of the streamedtitle are clear and useable. These formatted streams are delivered 190to the output device 103 where they are received and displayed 140 in amanner consistent with the display capabilities of the output device. Inalternative embodiments, the output device may prioritize the display ofthe streams 150. By way of example, and not by way of limitation, ifeach aspect of the UI data is contained in a separate stream, the outputdevice may prioritize the display of the streams in the order of (textdata, video data, map data, etc.) so that certain aspects of the displaymaintain clarity during times of network latency while others becomeless defined.

As an example, a ‘game’ stream for an online video game may be acompressed video stream. The UI stream for the game may not necessarilybe a video stream, but as noted above can be text, bitmap data, vectoredgraphics and other types of data. These types of data may be shared withthe output device 103. In some implementations, the output device 103may already have some of the data locally, e.g., as part of the data236′ in the memory 232′ or cache 244′. For example the fonts used torender text may already be available on the client. In someimplementations, the output device 103 may be able to configure the UPbased on user settings. In such an implementation the user could, e.g.,override bitmaps or fonts with whatever data is already stored on theoutput device 103 to create something like a game ‘mod’ which is oftenused in massive multiplayer online games (MMOs).

The illustration in FIG. 2 also provides an example of a second userrequesting to stream a media title from a server, and is provided toshow that the data may be streamed to various devices which each havetheir own respective data stream 140′ such that the data is received anddisplayed in a manner consistent with the display capabilities of amultitude of output devices.

It is emphasized that the example depicted in FIG. 2 is provided forpurposes of illustration only, in order to highlight certain aspects ofthe present disclosure. In practice, implementations of the presentdisclosure may factor in additional or alternative considerations notdepicted by the example of FIG. 2, and may be more complex than thesimplified scheme depicted in FIG. 2.

Certain implementations of aspects of the present disclosure includesystems configured for the dynamic adjustment of streamed content. Byway of example, and not by way of limitation, FIG. 3 depicts adistributed computing system that includes three devices 102, 103, and104, and the computing systems 102, 103, and 104 are configured totransfer data over a network in accordance with certain aspects of thepresent disclosure. In certain implementations, the device 102 may beconfigured to execute instructions that have one or more aspects incommon with those described with respect to FIG. 2 and/or FIG. 4. Incertain implementations, the devices 103 and 104 may be configured toexecute instructions that have one or more aspects in common with one ormore of those described above with respect to FIG. 2 or below withrespect to FIG. 5. Each of the devices 102, 103, and 104 may beconfigured with suitable software and/or hardware to implement variousaspects of the methods described herein. Each of the devices 102, 103,and 104 may be a server, an embedded system, mobile phone, personalcomputer, laptop computer, tablet computer, portable game device,workstation, game console, wearable devices such as a smart watches,“dongle” devices and the like. As used herein, the term “dongle device”refers to a device that plugs into other equipment, such as a computeror television, to add functionality to the equipment. Such functionalitymay include as copy protection, audio, video, games, data, or otherservices that are available only when the dongle is attached. Examplesof dongle devices include, but are not limited to internet contentstreaming devices and infrared remote control adapters available for“smart” mobile phones. The output devices 103, 104 have certaincomponents in common with the server 102. In FIG. 3 components of theserver 102 are denoted by reference numerals without primes,corresponding components of output device 103 are denoted by the samereference numerals with primes (′) and corresponding components ofoutput device 104 are denoted by the same reference numerals with doubleprimes (″).

In accordance with certain implementations, the device 102 may be aserver configured to format and provide media streaming data, and thedevices 103 and 104 may be an output device utilized by a user whowishes to display a media title. The server 102 may be configured tocreate and format media 188 and deliver the data 190 to an output device103 or 104 over a network 260, e.g., using an internet connection or alocal area network connection. It is noted that the network 260 need notbe an internet connection. In some implementations streaming may takeplace from one device at a given location to another device in thatlocation via a local area network.

Furthermore, even if the network 260 is a wide area network, it may beimplemented by technology other than the internet, such as, a cablenetwork.

The data streams 188 may be separately formatted for respective devices103 or 104 even if the media title requested is the same, as the devicecapabilities 186 of the respective output devices may vary. By way ofexample, and not by way of limitation, the data streams 188 may includestreams for video data, UI data, text, inventory data, map data, oraudio data. In some implementations certain types of data streams may becompressed to reduce the number of bits of data that need to betransmitted in the stream. In particular, video data and audio data arecommonly compressed prior to being transmitted.

Each of the devices 102, 103, and 104 may include one or more processorunits 231, 231′, 231″ which may be configured according to well-knownarchitectures, such as, e.g., single-core, dual-core, quad-core,multi-core, processor-coprocessor, cell processor, and the like. Eitherof the devices Each of the devices 102, 103, and 104 may also includeone or more memory units 232, 232′, 232″ (e.g., RAM, DRAM, ROM, and thelike). The processor unit 231, 231′, 231″ may execute one or moreprograms 233, 233′, 233″ which may be stored in the memory 232, and theprocessor 231, 231′, 231″ may be operatively coupled to the memory 232,232′, 232″, e.g., by accessing the memory via a data bus 250, 250′,250″. The memory unit 232, 231′, 231″ may include data 236, 236′, 236″and the processor unit 231, 231′, 231″ may utilize the data 236, 236′,236″ in implementing the program 233, 233′, 233″. The data 236, 236′,236″ for each of the devices 102, 103, and 104 may include, e.g., arequest for streaming data 120 transmitted from an output device 103 or104 to the server 102, and a specially formatted set of data streams 188for delivery 140 from the server 102 to an output device 103 or 104according to various aspects of the present disclosure. The program 233,233′, 233″ may include optionally instructions that, when executed by aprocessor, perform one or more operations associated with requestingstreaming data for a media title 120, determining the capabilities ofthe output device from which the request was received 186, creating andformatting two or more video streams which respectively contain videoand UI data 188 for display on the output device 103 or 104, orreceiving and displaying data streams formatted 140 to best utilize thedisplay capabilities of the output device 103 or 104; e.g., a methodhaving one or more features in common with the methods of FIGS. 2, 4,and/or 5. For example, the program 233, 233′, 233″ of the server 102 mayinclude instructions that, when executed by the processor 231, 231′,231″, cause the server to format and deliver streaming data to the atleast one recipient device 103 or 104, in accordance with aspects of theserver side of the method depicted in FIG. 2 and/or the sending ofstreaming data. In alternative embodiments, the program 233, 233′, 233″of the server 102 may include instructions that, when executed by theprocessor 231, 231′, 231″, cause the server to request displaycapability information from the at least one recipient device 103 or104, in accordance with aspects of the server side of the methoddepicted in FIG. 2 and/or the sending of streaming data. The program233, 233′, 233″ of the output device 103 or 104 may include instructionsthat, when executed by the processor 231, 231′, 231″, cause the outputdevice to request streaming data 120 that can then be provided by theserver 102 in accordance with aspects of the output device side of themethod depicted in FIG. 2 and/or the sending of streaming data. Inalternative embodiments, the program 233, 233′, 233″ of the outputdevice 103 or 103 may include instructions that, when executed by theprocessor 231, 231′, 231″, cause the output device to deliver displaycapability information 184 to the server 102 in accordance with aspectsof the output device side of the method depicted in FIG. 2 and/or thesending of streaming data.

Each of the devices 102, 103, and 104 may also include well-knownsupport circuits 240, 240′, 240″, such as input/output (I/O) circuits241, 241′, 241″ (which in the case of output devices 103, 104 may becoupled to a controller 245′, 245″), power supplies (P/S) 242, 242′,242″, a clock (CLK) 243, 243′, 243″, and cache 244, 244′, 244″, whichmay communicate with other components of the system, e.g., via the bus250, 250′, 250″. Each of the devices 102, 103, and 104 may optionallyinclude a mass storage device 234, 234′, 234″ such as a disk drive,CD-ROM drive, tape drive, flash memory, or the like, and the massstorage device 234, 234′, 234″ may store programs and/or data. Each ofthe devices 102, 103, and 104 may also optionally include a display unit237, 237′, 237″. The display unit 237, 237′, 237″ may be in the form ofa cathode ray tube (CRT), flat panel screen, touch screen, or otherdevice that displays text, numerals, graphical symbols, or other visualobjects. Each of the devices 102, 103, and 104 may also include a userinterface 206, 206′, 206″ to facilitate interaction between the device102/103 or 104 and a user. The user interface 206, 206′, 206″ mayinclude a keyboard, mouse, light pen, game control pad, touch interface,or other device. The user interface may also include an audio I/Odevice, such as a speaker and/or microphone.

A user may interact with either of the computer systems through the userinterface 206, 206′, 206″. By way of example, the server may 102 may bea cloud gaming server, and the output device 103 or 104 may be a cloudgaming client, and a video game user may interact with a video gameexecuted by the server 102 and streamed to the output device 104 throughthe user interface 206, 206′, 206″. Portions of the user interface 206,206′, 206″ may include a graphical user interface (GUI) that can bedisplayed on the display unit 237, 237′, 237″ in order to facilitateuser interaction with the system 102/103 or 104. The system 102/103 or104 may include a network interface 239, 239′, 239″, configured toenable the use of Wi-Fi, an Ethernet port, or other communicationmethods. The network interface 239, 239′, 239″ may incorporate suitablehardware, software, firmware or some combination thereof to facilitatecommunication via a telecommunications network 260, and may support datatransport using an unreliable protocol in accordance with certainaspects of the present disclosure. The network interface 239, 239′, 239″may be configured to implement wired or wireless communication overlocal area networks and wide area networks such as the Internet.

As shown in FIG. 4, a set of server side instructions 370 may beimplemented, e.g., by the server 102. The server instructions 370 may beformed on a nontransitory computer readable medium such as the memory232, 232′, 232″ or the mass storage device 234, 234′, 234″. The serverside instructions 370 may also be part of the process control program233, 233′, 233″. As indicated at 372 the server instructions 370 mayinclude instructions for requesting output device information,optionally including output device display capabilities 183 to bereceived from or more client devices 103 or 104 over a network 260. Theinstructions 370 may include instructions 374 for formatting two or morestreams of content data 188 based on the capabilities of the outputdevice. Thereafter, at 376 the instructions may include instructions fordelivering 190 the data streams to the output device 103 or 104 andsubsequently, at 378 may include instructions for delivering data streamformatting or priority information to the respective output device.

As shown in FIG. 5, a set of output device instructions 480 may beimplemented, e.g., by the output device 103 or 104. The output deviceinstructions 480 may be formed on a nontransitory computer readablemedium such as the memory 232, 232′, 232″ or the mass storage device234, 234′, 234″. The output device instructions 480 may also be part ofthe process control program 233, 233′, 233″. As indicated at 482, theinstructions 480 may include instructions for delivering output deviceinformation, optionally including output device display capabilities 183to be delivered by the or more client devices 103 or 104 over a network260. The instructions 470 may include instructions 474 for receiving 140two or more streams of content data 188 based on the capabilities of theoutput device. Thereafter, at 476 the instructions may includeinstructions for prioritizing 150 the display of the data streams on theoutput device 103 or 104 and subsequently, at 478 may includeinstructions for formatting text or other user interface (UI) such thatthe streamed content is displayed in a manner consistent with thedisplay capabilities of the output device 103 or 104.

The above components may be implemented in hardware, software, firmware,or some combination thereof.

Although the examples described above assume a game stream and a UIstream, which are combined on the client side, additional streams,example a video chat stream, may be included. Such additional streamsmay come peer-to-peer from other users' client devices and could becombined by the client devices. In some implementations, the program 233on the server 102 may be aware that there is a video chat stream betweenoutput devices 103 and 104 and could reserve resources for handling thisstream. For latency reasons the server 102 could coordinate apeer-to-peer (p2p) session to let the client devices 103, 104 combinethe video chat stream with the other streams. There may be types oflocal content which could be composited by the client devices.

Aspects of the present disclosure allow for formatting and streaming ofdata in different formats that are displayed together to be optimallydisplayed on different devices.

While the above is a complete description of the preferred embodiment ofthe present invention, it is possible to use various alternatives,modifications and equivalents. Therefore, the scope of the presentinvention should be determined not with reference to the abovedescription but should, instead, be determined with reference to theappended claims, along with their full scope of equivalents. Any featuredescribed herein, whether preferred or not, may be combined with anyother feature described herein, whether preferred or not. In the claimsthat follow, the indefinite article “a”, or “an” refers to a quantity ofone or more of the item following the article, except where expresslystated otherwise. The appended claims are not to be interpreted asincluding means-plus-function limitations, unless such a limitation isexplicitly recited in a given claim using the phrase “means for.”

What is claimed is:
 1. A non-transitory computer readable medium havingprocessor-executable instructions embodied therein, wherein execution ofthe instructions by a processor causes the processor to implement amethod, the method comprising: a) determining the display capabilitiesof an output device; b) formatting two or more data streams configuredfor display on the output device such that the content is displayed in amanner consistent with the display capabilities of the output device,wherein the two or more data streams include a data stream for userinterface data and a separate data stream for video data; and c)delivering the two or more streams of content data to the output device.2. The non-transitory computer readable medium of claim 1, wherein thetwo or more data streams include a data stream containing compresseddata.
 3. The non-transitory computer readable medium of claim 1, whereinthe two or more streams of content include a data stream containing datato be displayed as text.
 4. The non-transitory computer readable mediumof claim 1, wherein the two or more streams of content include a datastream containing data to be displayed as a heads-up-display (HUD). 5.The non-transitory computer readable medium of claim 1, wherein the twoor more data streams include a data stream containing data to bedisplayed as inventory information.
 6. The non-transitory computerreadable medium of claim 1, wherein the two or more data include a datastream containing data to be displayed map information.
 7. Thenon-transitory computer readable medium of claim 1, wherein the displayof one data stream is prioritized over the display of another.
 8. Thenon-transitory computer readable medium of claim 7, wherein theprioritization of display is determined by the quality of the networkconnection.
 9. The non-transitory computer readable medium of claim 1,wherein the display instructions are delivered by a server to a clientdevice platform used in combination with the output device.
 10. Thenon-transitory computer readable medium of claim 9, wherein the clientdevice platform is a gaming console.
 11. The non-transitory computerreadable medium of claim 9, wherein the client device platform is acomputer.
 12. The non-transitory computer readable medium of claim 9,wherein the client device platform is a cellular phone.
 13. Thenon-transitory computer readable medium of claim 9, wherein the clientdevice platform is a tablet.
 14. The non-transitory computer readablemedium of claim 9, wherein the client device platform is a hand-heldcomputing device.
 15. The non-transitory computer readable medium ofclaim 9, wherein the client device platform is a set top box.
 16. Thenon-transitory computer readable medium of claim 9, wherein the clientdevice platform is a telephonic system.
 17. The non-transitory computerreadable medium of claim 9, wherein the client device platform is adongle device.
 18. The non-transitory computer readable medium of claim9, wherein the client device is configured to query the output device todetermine the display capabilities of the output device and deliver thisinformation to the server.
 19. The non-transitory computer readablemedium of claim 1, wherein the display capability information alsoincludes a picture orientation of the output device.
 20. Thenon-transitory computer readable medium of claim 1, wherein the contentdata includes a video portion of a video game.
 21. The non-transitorycomputer readable medium of claim 1, wherein the content data is astream of a computer program session running on a remote machine. 22.The non-transitory computer readable medium of claim 1, wherein theoutput device is a television.
 23. The non-transitory computer readablemedium of claim 1, wherein the output device is a projector.
 24. Thenon-transitory computer readable medium of claim 1, wherein the outputdevice and the client device platform are the same device.
 25. Thenon-transitory computer readable medium of claim 1, wherein b) includesformatting the video data with an application used to generate videoframes for the data stream for video data.
 26. The non-transitorycomputer readable medium of claim 1, wherein the two or more streamsinclude a game stream and a user interface stream.
 27. Thenon-transitory computer readable medium of claim 26, wherein the two ormore streams include a game stream, a user interface stream and one ormore additional streams.
 28. The non-transitory computer readable mediumof claim 26, wherein the two or more streams include a game stream, auser interface stream and a video chat stream between a first clientdevice and a second client device.
 29. The non-transitory computerreadable medium of claim 26, wherein the two or more streams include agame stream, a user interface stream and a video chat stream between afirst client device and a second client device, wherein the methodfurther comprises coordinating a peer-to-peer session between the firstand second client devices to let the first and second client devicescombine the video chat stream with the game stream and the userinterface stream.
 30. On a server configured to operate on a network, amethod, comprising: a) determining the display capabilities of an outputdevice; b) formatting two or more data streams configured for display onthe output device such that the content is displayed in a mannerconsistent with the display capabilities of the output device, whereinthe two or more data streams include a data stream for user interfacedata and a separate data stream for video data.
 31. A system comprising:a processor, and a memory coupled to the processor; wherein theprocessor is configured to perform a method, the method comprising: a)determining the display capabilities of an output device; b) formattingtwo or more data streams configured for display on the output devicesuch that the content is displayed in a manner consistent with thedisplay capabilities of the output device, wherein the two or more datastreams include a data stream for user interface data and a separatedata stream for video data; and c) delivering the two or more streams ofcontent data to the output device.